Systems and methods for autonomous vehicle operator vigilance management

ABSTRACT

Systems and methods for managing autonomous vehicle operator vigilance are provided. A method can include determining, by a computing system comprising one or more processors, a first vigilance prompt. The first vigilance prompt can be included in a plurality of vigilance prompts. Each of the plurality of vigilance prompts can be different from each other vigilance prompt. Each vigilance prompt can be a prompt for the vehicle operator to perform a particular interaction with the autonomous vehicle. The method can further include providing, by the computing system, the first vigilance prompt to a vehicle operator of an autonomous vehicle. The method can further include receiving, by the computing system, a first response from the vehicle operator in response to the first vigilance prompt. The method can further include determining, by the computing system, a vehicle operator vigilance level based at least in part on the first response.

PRIORITY CLAIM

The present application is based on and claims benefit of U.S.Provisional Application 62/711,044 having a filing date of Jul. 27,2018, which is incorporated by reference herein.

FIELD

The present disclosure relates generally to devices, systems, andmethods for managing autonomous vehicle operator vigilance.

BACKGROUND

An autonomous vehicle is a vehicle that is capable of sensing itsenvironment and navigating with minimal or no human input. Inparticular, an autonomous vehicle can observe its surroundingenvironment using a variety of sensors and can attempt to comprehend theenvironment by performing various processing techniques on datacollected by the sensors. Given knowledge of its surroundingenvironment, the autonomous vehicle can identify an appropriate motionpath through such surrounding environment.

SUMMARY

Aspects and advantages of embodiments of the present disclosure will beset forth in part in the following description, or can be learned fromthe description, or can be learned through practice of the embodiments.

One example aspect of the present disclosure is directed to a method fordetermining a vigilance level of a vehicle operator in an autonomousvehicle. The method can include determining, by a computing systemcomprising one or more processors, a first vigilance prompt. The firstvigilance prompt can be included in a plurality of vigilance prompts.Each of the plurality of vigilance prompts can be different from eachother vigilance prompt. Each vigilance prompt can be a prompt for thevehicle operator to perform a particular interaction with the autonomousvehicle. The method can further include providing, by the computingsystem, the first vigilance prompt to a vehicle operator of anautonomous vehicle. The method can further include receiving, by thecomputing system, a first response from the vehicle operator in responseto the first vigilance prompt. The method can further includedetermining, by the computing system, a vehicle operator vigilance levelbased at least in part on the first response.

Another example aspect of the present disclosure is directed to anautonomous vehicle operator vigilance management system. The autonomousvehicle operator vigilance management system can include one or moreprocessors and one or more tangible, non-transitory, computer readablemedia that collectively store instructions that when executed by the oneor more processors cause the computing system to perform operations. Theoperations can include determining that a vehicle operator is ready tobe provided a first vigilance prompt. The first vigilance prompt can beincluded in a plurality of vigilance prompts. Each of the plurality ofvigilance prompts can be different from each other vigilance prompt.Each vigilance prompt can be a prompt for the vehicle operator toperform a particular interaction with the autonomous vehicle. Theoperations can further include determining the first vigilance prompt.The operations can further include providing the first vigilance promptto the vehicle operator of the autonomous vehicle. The operations canfurther include receiving a first response from the vehicle operator inresponse to the first vigilance prompt.

Another example aspect of the present disclosure is directed to anautonomous vehicle configured to operate in a testing environment. Theautonomous vehicle can include a touch-sensitive display screen, aspeaker device, and one or more of vehicle operator input devices. Theone or more vehicle operator input devices can include one or more of: abutton, a shifter paddle, a turn indicator, a windshield wiper, amicrophone, or the touch-sensitive display screen. The autonomousvehicle can further include a vehicle operator testing normalizationsystem. The vehicle operator testing normalization system can includeone or more processors; and one or more tangible, non-transitory,computer readable media that collectively store instructions that whenexecuted by the one or more processors cause the computing system toperform operations. The operations can include providing a firstvigilance prompt to a vehicle operator of the autonomous vehicle via atleast one of the touch-sensitive display screen or the speaker device.The first vigilance prompt can be a prompt for the vehicle operator toperform a particular interaction with at least one vehicle operatorinput device. The operations can further include receiving a firstresponse from the vehicle operator in response to the first vigilanceprompt via the at least one vehicle operator device. The operations canfurther include determining a second vigilance prompt based at least inpart on the first response. The second vigilance prompt can include acognitive loading component. The operations can further includeproviding the second vigilance prompt to the vehicle operator of theautonomous vehicle.

Other aspects of the present disclosure are directed to various systems,apparatuses, non-transitory computer-readable media, vehicles, andcomputing devices.

These and other features, aspects, and advantages of various embodimentsof the present disclosure will become better understood with referenceto the following description and appended claims. The accompanyingdrawings, which are incorporated in and constitute a part of thisspecification, illustrate example embodiments of the present disclosureand, together with the description, serve to explain the relatedprinciples.

BRIEF DESCRIPTION OF THE DRAWINGS

Detailed discussion of embodiments directed to one of ordinary skill inthe art is set forth in the specification, which makes reference to theappended figures, in which:

FIG. 1 depicts an example autonomous vehicle computing system accordingto example aspects of the present disclosure;

FIG. 2 depicts an example vehicle operator's perspective of an insidecab portion of an autonomous vehicle according to example aspects of thepresent disclosure;

FIG. 3 depicts an example vigilance management system configurationaccording to example aspects of the present disclosure;

FIG. 4 depicts an example vigilance management system configurationaccording to example aspects of the present disclosure;

FIG. 5 depicts an example vehicle operator vigilance management processaccording to example aspects of the present disclosure;

FIG. 6 depicts an example method according to example aspects of thepresent disclosure;

FIG. 7 depicts an example method according to example aspects of thepresent disclosure; and

FIG. 8 depicts example system components according to example aspects ofthe present disclosure.

DETAILED DESCRIPTION

Example aspects of the present disclosure are directed to systems andmethods for improving vehicle operator vigilance during autonomousvehicle driving sessions. An autonomous vehicle can drive, navigate,operate, etc. with minimal and/or no interaction from a human driver toprovide a vehicle service. By way of example, an autonomous vehicle canbe configured to provide transportation and/or other services, such astransporting a passenger from a first location to a second location. Insome applications, a vehicle operator (e.g., driver) may be present inthe vehicle, such as to take over manual control of the vehicle when theautonomous vehicle is operated in a testing environment. For example,during an autonomous operation testing session, periodic faults can beinjected into an autonomous vehicle's autonomy system to test how theautonomous vehicle responds to certain scenarios. During such faults,the vehicle operator can provide backup control of the autonomousvehicle, such as by assuming manual control of the vehicle. However,during the course of such driving sessions, vehicle operators may becomefatigued due to the infrequency of faults.

The systems and methods of the present disclosure can help determine avehicle operator vigilance level and manage the vehicle operatorvigilance level by engaging the vehicle operator with vigilance prompts.For example, a computing system can determine a vigilance prompt to beprovided to a vehicle operator from a plurality of different vigilanceprompts. Each vigilance prompt in the plurality can be, for example, aprompt for the vehicle operator to perform a particular interaction withthe autonomous vehicle. The vigilance prompts can be specific tasksselected to improve vehicle operator responsiveness through alertness,interest, and motivation. For example, in some implementations, avigilance prompt can be a visual cue displayed on a display screen ofthe autonomous vehicle or an audio queue played by a speaker of theautonomous vehicle. For example, a text command can be displayed on thedisplay screen indicating a particular interaction, such as to press abutton, touch the display screen, press a particular shifter paddle(e.g., left or right), actuate a turn indicator, actuate the windshieldwipers, provide a verbal response, or other particular interaction. Thecomputing system can then provide the vigilance prompt to the vehicleoperator and receive a response from the vehicle operator when thevehicle operator performs the particular task in response to thevigilance prompt. For example, in response to receiving the prompt, thevehicle operator can perform the particular interaction, such aspressing the particular shifter paddle, etc., which can be received bythe computing system as a response. The computing system can thendetermine a vehicle operator vigilance level based at least in part onthe response. For example, the vehicle operator vigilance level can bedetermined based on an accuracy of the response (e.g., whether thevehicle operator performed the correct task) and/or a response time(e.g., how long it took the vehicle operator to provide the response).The computing system can then determine a vehicle operator managementaction based on the vigilance level, and implement the managementaction. For example, if the vehicle operator vigilance level is low,subsequent vigilance prompts can be provided at more frequent intervalsin order to help maintain a vehicle operator's alertness.

More particularly, an autonomous vehicle (e.g., ground-based vehicle,etc.) can include various systems and devices configured to control theoperation of the vehicle. For example, an autonomous vehicle can includean onboard vehicle computing system (e.g., located on or within theautonomous vehicle) that is configured to operate the autonomousvehicle. The vehicle computing system can obtain sensor data fromsensor(s) onboard the vehicle (e.g., cameras, LIDAR, RADAR, etc.),attempt to comprehend the vehicle's surrounding environment byperforming various processing techniques on the sensor data, andgenerate an appropriate motion plan through the vehicle's surroundingenvironment. For example, the sensor data can be used in a processingpipeline that includes the detection of objects proximate to theautonomous vehicle, object motion prediction, and vehicle motionplanning. For example, a motion plan can be determined by the vehiclecomputing system, and the vehicle can be controlled by a vehiclecontroller to initiate travel in accordance with the motion plan. Theautonomous vehicle can also include one or more display screens, such astouch-sensitive interactive display screens, speakers, or other devicesconfigured to provide a vehicle operator with vigilance prompts.Further, the autonomous vehicle can include various vehicle inputdevices, such as buttons, paddle shifters, microphones, turn indicators,windshield wipers, motion detectors (e.g., for gesture tracking), etc.The vehicle operator can perform various interactions with theautonomous vehicle through the various vehicle input devices.

In some implementations, the computing system can determine that thevehicle operator is ready to be provided a vigilance prompt. Forexample, the computing system can access a motion plan to determine thatthe autonomous vehicle is operating within an authorized area, and thatthe autonomous vehicle does not have any pending turns or lane changeswithin a testing period. For example, the testing period can be aparticular time period (e.g., five seconds, ten seconds, etc.), and themotion plan can indicate that the vehicle is to travel on a straightawaypath during the testing period. In some implementations, the computingsystem can determine that the vehicle operator is ready to be provided avigilance prompt when the vehicle operator has not been provided aprevious vigilance prompt within a threshold time period. For example,the vehicle operator can be provided with a vigilance prompt ifsufficient time has elapsed since the vehicle operator was last provideda vigilance prompt (e.g., ten minutes).

The computing system can then determine a vigilance prompt to provide tothe vehicle operator from a plurality of different vigilance prompts.The vigilance prompt can be a prompt to perform a particular interactionwith the autonomous vehicle. The vigilance prompts in the plurality canbe uncomplicated tasks which are sufficient to engage a vehicle operatorwithout compromising the vehicle operator's ability to assume manualcontrol of the autonomous vehicle. For example, the vehicle operator canbe provided periodic vigilance prompts at regular or random intervals inorder to increase vehicle operator alertness. Additionally, theplurality of vigilance prompts can include a variety of differentvigilance prompts, which can further help to stimulate the vehicleoperator. For example, if only a single type of vigilance prompt isused, the vehicle operator may become conditioned to perform theassociated single specific task without improving the vehicle operator'salertness. The variety of vigilance prompts according to example aspectsof the present disclosure, however, can prevent and overcome suchconditioning of vehicle operators. For example, in some implementations,the computing system can access a list of possible vigilance prompts andrandomly select a vigilance prompt. In some implementations, thecomputing system can select a vigilance prompt from the plurality thatis different from a most recently provided vigilance prompt. In someimplementations, the computing system can use complex patterns,probabilities, or other suitable methods to select the vigilance prompt.

The computing system can then provide the vigilance prompt to thevehicle operator. For example, in some implementations, an autonomousvehicle can include a display screen, such as a touch-sensitive displayscreen. The computing system can provide a vigilance prompt bydisplaying a visual cue on the display screen. In variousimplementations, the visual cue can implicitly or explicitly prompt thevehicle operator to perform a particular interaction. For example, insome implementations, the visual cue can be a change from a first colorto a second color (e.g., red to green). The vehicle operator can beinformed and/or trained to perform a particular interaction with theautonomous vehicle upon the color change. For example, the vehicleoperator can press a button or touch the display screen in response toseeing the color change.

In some implementations, the visual cue can be a displayed icon (e.g.,an icon showing a left shifter paddle push). In some implementations,the displayed icon can include an animation of the particularinteraction the vehicle operator is to perform. In some implementations,the visual cue can be a text command (e.g., “left turn indicator”). Inresponse, the vehicle operator can perform the particular interactionassociated with the vigilance prompt with the autonomous vehicle. Othersuitable visual cues can similarly be provided.

In some implementations, the vigilance prompt can be an audio cue playedby a speaker of the autonomous vehicle. For example, the computingsystem can play a verbal command over one or more speakers of theautonomous vehicle. The verbal command can describe the particularinteraction the vehicle operator is to perform (e.g., “activatewindshield wipers”). In response, the vehicle operator can perform theparticular interaction. Other suitable audio cues can similarly beprovided.

In some implementations, each vigilance prompt can have an associatedparticular interaction with a component on or near the steering wheel ofthe autonomous vehicle (e.g., turn indicator, shifter paddle, windshieldwiper actuator, etc.), which can allow for the vehicle operator's handsto be maintained in close proximity to the steering wheel whileresponding to the vigilance prompt. The operator's hands can activatethe component in a manner that is typical with that component and, thus,not distracting from the operator's typical vehicle operation. In suchan implementation, the vehicle operator's ability to assume manualcontrol of the autonomous vehicle can be increased due to the vehicleoperator's hands being in close proximity to the steering wheel.

In some implementations, the autonomous vehicle can include one or moremicrophones configured to receive verbal responses from the vehicleoperator. For example, a display screen can display text for the vehicleoperator to read out loud, such as letter or number. In someimplementations, a vigilance prompt can be a simple question to whichthe vehicle operator is to provide a verbal answer. In response to suchvigilance prompts, the vehicle operator can audibly read the text orrespond to the question, which can be received by the microphone andprovided to the computing system. For example, a computing system can beconfigured to recognize speech, which can then be used to determine anaccuracy of the response.

The computing system can then determine a vigilance level based at leastin part on the vehicle operator's response. In some implementations, thevigilance level can be determined based on an accuracy of the vehicleoperator's response, such as whether the vehicle operator performed theparticular interaction in the vigilance prompt correctly (e.g., pushedthe correct shifter paddle). In some implementations, the vigilancelevel can be determined based on a response time, such as the time ittook for the vehicle operator to perform the particular interactionfollowing the vigilance prompt. For example, the computing system canreference lookup tables for particular vigilance prompts, such asacceptable response times and/or acceptable accuracy rates for theparticular vigilance prompts. In some implementations, the vigilancelevel can be expressed as a numerical value (e.g., on a scale), apercentage, or other suitable format. In some implementations, thevigilance level can be expressed as category (e.g. pass/fail,high/medium/low, responsive/nonresponsive, etc.). In someimplementations, the vigilance level can be determined based on aplurality of vehicle operator responses, such as a plurality of the mostrecently received responses (e.g., a mean, median, rolling window,etc.). In this way, the vigilance level determined by the computingsystem can be indicative of the overall alertness and vigilance of thevehicle operator, both historically as well as at a particular time. Insome implementations, each vehicle operator can have an associatedprofile, and respective vehicle operator vigilance levels can be trackedover time, such as over a plurality of autonomous driving sessions.

In some implementations, the computing system can determine a vehicleoperator management action based at least in part on the vigilancelevel. For example, in some implementations, the vehicle operatormanagement action can be to provide one or more subsequent vigilanceprompts to the vehicle operator. For example, if a vehicle operatorvigilance level is low, the computing system can determine a time delaybased at least in part on the vehicle operator vigilance level. Forexample, for a low vigilance level, the time delay for a secondvigilance prompt can be a short time delay, such as a five minute delayas compared to a 10 minute delay for a middle vigilance level or a 15minute delay for a high vigilance level. Stated differently, thecomputing system can use the vehicle operator vigilance level todetermine (e.g., adjust) the frequency of one or more subsequentvigilance prompts. By providing subsequent vigilance prompts at a higherfrequency, the vehicle operator's vigilance can be increased bystimulating the vehicle operator more frequently. Similarly, for vehicleoperators with a high vigilance level, the frequency of subsequentvigilance prompts can, in some instances, be reduced, as the vehicleoperator or may not require additional stimulation to maintain his/hervigilance.

In some implementations, the vehicle operator management action can beto determine (e.g., adjust) the type of one or more subsequent vigilanceprompts. For example, certain vigilance prompts and their associatedparticular interactions may be better at stimulating and engaging thevehicle operator, and therefore more effective in maintaining thevehicle operator's vigilance. For example, in some implementations,analysis of a particular vehicle operator's profile may indicate thattouching a display screen in response to a color change is moreeffective at increasing the vehicle operator vigilance level thanpressing a button in response to an icon display. Accordingly, if thevehicle operator vigilance level is low, the computing system canincrease the frequency of the display screen touch vigilance promptrelative to the button press vigilance prompt. Similarly, for somevehicle operators, audio cues may be more effective in maintaining avehicle operator vigilance level than visual cues, or vice-versa.Accordingly, if a vehicle operator vigilance level is low, the computingsystem can provide audio cues rather than visual cues, or vice-versa. Inthis way, a computing system can determine a vigilance prompt based atleast in part on a vehicle operator vigilance level.

Similarly, in some implementations, the type of vigilance prompt caninclude both a visual cue and an audio cue. For example, if a vehicleoperator vigilance level is low, such as following several consecutivevigilance prompts, the computing system can escalate the type ofvigilance prompt by concurrently providing both an audio cue and avisual cue with each vigilance prompt. By providing both a visual cueand an audio cue, the vehicle operator may be stimulated, and thereforehave improved vigilance.

In some implementations, the vehicle operator can be provided feedbackon the vehicle operator vigilance level. For example, the computingsystem can display or audibly play an indicator of the vehicle operatorvigilance level (e.g., a numerical value, score, percentage, accuracy,response time, or category, such as pass/fail, high/low, etc.). In someimplementations, a vehicle operator vigilance level for a particularvigilance prompt can be provided, such as an indication of the accuracyor response time following a response to a particular vigilance prompt.In some implementations, the vehicle operator vigilance level can tracka vehicle operator's vigilance over a plurality of vigilance prompts,which can indicate whether a vehicle operator's vigilance is improvingor decreasing. By providing the vehicle operator with feedback on thevehicle operator vigilance level, the vehicle operator's interest andmotivation to perform well may cause the vehicle operator to become morevigilant.

In some implementations, the vehicle operator management action caninclude operating the autonomous vehicle to a safe state in whichautonomous operation is disabled. For example, if a vehicle operator'svigilance level is too low, such as following a failure to respond toone or more consecutive vigilance prompts, the computing system canimplement a motion plan to operate the vehicle to a safe state, such asnavigating the autonomous vehicle to a stop in a parking lot, andautonomous operation can be disabled. For example, the vehicle operatorcan be prevented from causing the autonomous vehicle to enter anautonomous operation mode until a reset has occurred, such as after thevehicle operator has been contacted by an external vehicle operatormanagement system. Further, the vehicle operator may only be able tooperate the autonomous vehicle in a manual operating mode until thereset has occurred.

Similarly, in some implementations, the autonomous vehicle may beconfigured to operate autonomously for one or more discrete portions ofa motion plan, and at the end of each portion, the vehicle operator maybe required to engage or reengage autonomous operation before theautonomous vehicle proceeds to initiate travel in accordance with themotion plan. If the vehicle operator vigilance level is too low, in someimplementations, the vehicle operator may be prevented from causing theautonomous vehicle to enter an autonomous operating mode.

In some implementations, a vehicle operator management action caninclude contacting an external vehicle operator management system. Forexample, an autonomous vehicle can include a communication systemconfigured to communicate with an external vehicle operator managementsystem. The external vehicle operator management system can be remotefrom the vehicle, and can provide various services such as securityservices, emergency services, roadside assistance, remote diagnostics,and/or communication services, such as with an emergency operator. Insome implementations, the external vehicle operator management systemcan assist the vehicle operator and/or remotely diagnose one or moreissues with the autonomous vehicle. For example, if a vehicle operatorvigilance level is low, a remote operator can ask the vehicle operatorif there is anything in particular affecting the vehicle operator'sability to respond to vigilance prompts. For example, external weatherconditions or traffic conditions may impair a vehicle operator's abilityto respond to vigilance prompts due to the vehicle operator paying closeattention to the conditions in which the autonomous vehicle isoperating. Similarly, a vehicle operator input device, such as amicrophone, may be malfunctioning, and the external vehicle operatormanagement system may be able to remotely diagnose such malfunctions. Insuch situations, the external vehicle operator management system candetermine that the vehicle operator's vigilance level should not preventthe vehicle operator from causing the vehicle to enter or continueoperating in an autonomous operating mode. Similarly, the externalvehicle operator management system can reset or clear any disablement ofautonomous operation, such as when the autonomous vehicle has beenoperated to a safe state or the vehicle operator has been prevented fromcausing the vehicle to enter an autonomous operating mode.

In some implementations, the vehicle operator management action caninclude logging the vehicle operator vigilance level, such as for lateranalysis. For example, the computing system can locally store dataindicative of the vehicle operator vigilance level, such as individualresponses, response accuracy levels, response times, or other datarelated to or indicative of the vehicle operator vigilance level. Thecomputing system can further be configured to communicate the loggeddata, such as over a communications network, to an external computingsystem. In some implementations, analysis can be performed on the logdata in order to identify ways to maintain and/or improve a vehicleoperate or vigilance level. For example, machine learned models can beused to identify effective vigilance prompts, both on an individuallevel (e.g., a particular vehicle operator profile) or over a pluralityof vehicle operators (e.g., aggregated vehicle operator profiles).

According to another example aspect of the present disclosure, in someimplementations, an autonomous vehicle can be configured to operate in anon-public testing environment, such as on an isolated testing track.For example, in order to test how an autonomous vehicle responds toparticular operating conditions, periodic faults can be injected into avehicle's autonomy system, and the vehicle's response can be logged foranalysis. For example, a plurality of faults can be queued up to beinjected at regular or random intervals over the course of an autonomousvehicle testing session (e.g., every 10-20 minutes). In such anenvironment, a vehicle operator may be present to assume manual controlof the autonomous vehicle when such a fault occurs. However, individualvehicle operators may have different response capabilities. For example,some vehicle operators may maintain a hyper-vigilant state for anextended period of time following a fault, whereas other vehicleoperators may return to a normal resting state in the same period oftime. Vehicle operators in a hyper-vigilant state may respond much morequickly than vehicle operators in a normal resting state. Thus,individual vehicle operator performance can impact testing results whereoutcomes are heavily dependent on the quality of vehicle operatorreactions.

To address this, in some implementations, the computing system can beconfigured to determine a cognitive loading component based on a vehicleoperator vigilance level. The cognitive loading component can be, forexample, a cognitive load aspect of a vigilance prompt which requiresthe vehicle operator to perform a more or less complicated task. Forexample, rather than pushing a button, the vehicle operator may berequired to press a sequence of buttons, solve a mathematical questionand provide a verbal response, or perform some other cognitive task.Similarly, in some implementations, the variety of vigilance prompts andthe amount of time between the vigilance prompts can be increased orreduced. The cognitive loading component can be used to normalizevehicle operator response times. For example, reduced cognitive loadingcomponents can improve vehicle operator response times, while increasedcognitive loading components can slow vehicle operator response times.In this way, individual vehicle operator performance can be normalized,which can help prevent testing results from being skewed byhyper-vigilant vehicle operators.

The systems and methods described herein provide a number of technicaleffects and benefits. More particularly, the systems and methods of thepresent disclosure provide improved techniques for determining andmanaging vehicle operator vigilance in autonomous vehicles. To do so,aspects of the present disclosure allow a computing system to determineand provide a vehicle operator with a variety of vigilance promptsconfigured to stimulate and engage the vehicle operator. Further, thecomputing system can determine a vigilance level of a vehicle operator,and implement a variety of management actions to improve vehicleoperator vigilance. In this way, individual vehicle operatorresponsiveness can be improved through alertness, interest, andmotivation. In turn, vehicle operators can respond to autonomous vehiclefaults or other scenarios more quickly, thereby enhancing public safety.

Example aspects of the present disclosure can provide an improvement tovehicle computing technology, such as autonomous vehicle computingtechnology. For instance, the systems and methods of the presentdisclosure provide an improved approach to managing vehicle operatorvigilance. For example, a computing system (e.g., an onboard computingsystem of an autonomous vehicle) can determine a vehicle operator isready to receive a vigilance prompt. For example, the computing systemcan determine that the autonomous vehicle is operating within anauthorized area without any pending turns or lane changes within atesting period, and that the vehicle operator has not been provided aprevious vigilance prompt within a threshold time period. The computingsystem can then determine a particular vigilance prompt to be providedfrom a plurality of different vigilance prompts. Each vigilance promptcan be a prompt for the vehicle operator to perform a particularinteraction with the autonomous vehicle, and can be provided by visualor audio cues. The computing system can provide the vigilance prompt,and receive a response to the vigilance prompt from the vehicleoperator. The computing system can determine a vigilance level based atleast in part on the vehicle operator response. For example, the vehicleoperator vigilance level can be determined based at least in part on anaccuracy and/or a response time of the response. The computing systemcan then determine the vehicle operator management action based at leastin part on the vehicle operator vigilance level. For example, thecomputing system can adjust a frequency of one or more subsequentvigilance prompts, adjust a type of one or more subsequent vigilanceprompts, provide the vehicle operator with vigilance level feedback,operate the autonomous vehicle to a safe state in which autonomousoperation is disabled, prevent the vehicle operator from causing theautonomous vehicle to enter an autonomous operating mode, contact anexternal vehicle operator management system, or log the vehicle operatorvigilance level. This can efficiently determine and improve real-timevehicle operator vigilance using existing infrastructure of anautonomous vehicle.

With reference now to the FIGS., example aspects of the presentdisclosure will be discussed in further detail. FIG. 1 illustrates anexample vehicle computing system 100 according to example embodiments ofthe present disclosure. The vehicle computing system 100 can beassociated with an autonomous vehicle 105. The vehicle computing system100 can be located onboard (e.g., included on and/or within) theautonomous vehicle 105.

The autonomous vehicle 105 incorporating the vehicle computing system100 can be various types of vehicles. For instance, the autonomousvehicle 105 can be a ground-based autonomous vehicle such as anautonomous car, autonomous truck, autonomous bus, etc. The autonomousvehicle 105 can be an air-based autonomous vehicle (e.g., airplane,helicopter, or other aircraft) or other types of vehicles (e.g.,watercraft, etc.). The autonomous vehicle 105 can drive, navigate,operate, etc. with minimal and/or no interaction from a human operator(e.g., driver). In some implementations, a human operator can be omittedfrom the autonomous vehicle 105 (and/or also omitted from remote controlof the autonomous vehicle 105). In some implementations, a humanoperator (also referred to as a vehicle operator) can be included in theautonomous vehicle 105.

In some implementations, the autonomous vehicle 105 can be configured tooperate in a plurality of operating modes. The autonomous vehicle 105can be configured to operate in a fully autonomous (e.g., self-driving)operating mode in which the autonomous vehicle 105 is controllablewithout user input (e.g., can drive and navigate with no input from avehicle operator present in the autonomous vehicle 105 and/or remotefrom the autonomous vehicle 105). The autonomous vehicle 105 can operatein a semi-autonomous operating mode in which the autonomous vehicle 105can operate with some input from a vehicle operator present in theautonomous vehicle 105 (and/or a human operator that is remote from theautonomous vehicle 105). The autonomous vehicle 105 can enter into amanual operating mode in which the autonomous vehicle 105 is fullycontrollable by a vehicle operator (e.g., human driver, pilot, etc.) andcan be prohibited and/or disabled (e.g., temporary, permanently, etc.)from performing autonomous navigation (e.g., autonomous driving). Insome implementations, the autonomous vehicle 105 can implement vehicleoperating assistance technology (e.g., collision mitigation system,power assist steering, etc.) while in the manual operating mode to helpassist the vehicle operator of the autonomous vehicle 105.

The operating modes of the autonomous vehicle 105 can be stored in amemory onboard the autonomous vehicle 105. For example, the operatingmodes can be defined by an operating mode data structure (e.g., rule,list, table, etc.) that indicates one or more operating parameters forthe autonomous vehicle 105, while in the particular operating mode. Forexample, an operating mode data structure can indicate that theautonomous vehicle 105 is to autonomously plan its motion when in thefully autonomous operating mode. The vehicle computing system 100 canaccess the memory when implementing an operating mode.

The operating mode of the autonomous vehicle 105 can be adjusted in avariety of manners. For example, the operating mode of the autonomousvehicle 105 can be selected remotely, off-board the autonomous vehicle105. For example, a remote computing system (e.g., of a vehicle providerand/or service entity associated with the autonomous vehicle 105) cancommunicate data to the autonomous vehicle 105 instructing theautonomous vehicle 105 to enter into, exit from, maintain, etc. anoperating mode. In some implementations, the remote computing system canbe an emergency vehicle operator management system, as disclosed herein.By way of example, such data communicated to an autonomous vehicle 105by a remote computing system can instruct the autonomous vehicle 105 toenter into the fully autonomous operating mode. In some implementations,the operating mode of the autonomous vehicle 105 can be set onboardand/or near the autonomous vehicle 105. For example, the vehiclecomputing system 100 can automatically determine when and where theautonomous vehicle 105 is to enter, change, maintain, etc. a particularoperating mode (e.g., without user input). Additionally, oralternatively, the operating mode of the autonomous vehicle 105 can bemanually selected via one or more interfaces located onboard theautonomous vehicle 105 (e.g., key switch, button, etc.) and/orassociated with a computing device proximate to the autonomous vehicle105 (e.g., a tablet operated by authorized personnel located near theautonomous vehicle 105). In some implementations, the operating mode ofthe autonomous vehicle 105 can be adjusted by manipulating a series ofinterfaces in a particular order to cause the autonomous vehicle 105 toenter into a particular operating mode.

The vehicle computing system 100 can include one or more computingdevices located onboard the autonomous vehicle 105. For example, thecomputing device(s) can be located on and/or within the autonomousvehicle 105. The computing device(s) can include various components forperforming various operations and functions. For instance, the computingdevice(s) can include one or more processors and one or more tangible,non-transitory, computer readable media (e.g., memory devices, etc.).The one or more tangible, non-transitory, computer readable media canstore instructions that when executed by the one or more processorscause the autonomous vehicle 105 (e.g., its computing system, one ormore processors, etc.) to perform operations and functions, such asthose described herein for managing a vehicle operator's vigilance.

The autonomous vehicle 105 can include a communications system 120configured to allow the vehicle computing system 100 (and its computingdevice(s)) to communicate with other computing devices. The vehiclecomputing system 100 can use the communications system 120 tocommunicate with one or more computing device(s) that are remote fromthe autonomous vehicle 105 over one or more networks (e.g., via one ormore wireless signal connections). In some implementations, thecommunications system 120 can allow communication among one or more ofthe system(s) on-board the autonomous vehicle 105. The communicationssystem 120 can include any suitable components for interfacing with oneor more network(s), including, for example, transmitters, receivers,ports, controllers, antennas, and/or other suitable components that canhelp facilitate communication.

As shown in FIG. 1, the autonomous vehicle 105 can include one or morevehicle sensors 125, an autonomy computing system 130, one or morevehicle control systems 135, and other systems, as described herein. Oneor more of these systems can be configured to communicate with oneanother via a communication channel. The communication channel caninclude one or more data buses (e.g., controller area network (CAN)),on-board diagnostics connector (e.g., OBD-II), and/or a combination ofwired and/or wireless communication links. The onboard systems can sendand/or receive data, messages, signals, etc. amongst one another via thecommunication channel.

The vehicle sensor(s) 125 can be configured to acquire sensor data 140.This can include sensor data associated with the surrounding environmentof the autonomous vehicle 105. For instance, the sensor data 140 canacquire image and/or other data within a field of view of one or more ofthe vehicle sensor(s) 125. The vehicle sensor(s) 125 can include a LightDetection and Ranging (LIDAR) system, a Radio Detection and Ranging(RADAR) system, one or more cameras (e.g., visible spectrum cameras,infrared cameras, etc.), motion sensors, and/or other types of imagingcapture devices and/or sensors. The sensor data 140 can include imagedata, radar data, LIDAR data, and/or other data acquired by the vehiclesensor(s) 125. The autonomous vehicle 105 can also include other sensorsconfigured to acquire data associated with the autonomous vehicle 105.For example, the autonomous vehicle 105 can include inertial measurementunit(s), wheel odometry devices, and/or other sensors.

In some implementations, the sensor data 140 can be indicative of one ormore objects within the surrounding environment of the autonomousvehicle 105. The object(s) can include, for example, vehicles,pedestrians, bicycles, and/or other objects. The object(s) can belocated in front of, to the rear of, to the side of the autonomousvehicle 105, etc. The sensor data 140 can be indicative of locationsassociated with the object(s) within the surrounding environment of theautonomous vehicle 105 at one or more times. The vehicle sensor(s) 125can provide the sensor data 140 to the autonomy computing system 130.

In addition to the sensor data 140, the autonomy computing system 130can retrieve or otherwise obtain map data 145. The map data 145 canprovide information about the surrounding environment of the autonomousvehicle 105. In some implementations, an autonomous vehicle 105 canobtain detailed map data that provides information regarding: theidentity and location of different roadways, road segments, buildings,or other items or objects (e.g., lampposts, crosswalks, curbing, etc.);the location and directions of traffic lanes (e.g., the location anddirection of a parking lane, a turning lane, a bicycle lane, or otherlanes within a particular roadway or other travel way and/or one or moreboundary markings associated therewith); traffic control data (e.g., thelocation and instructions of signage, traffic lights, or other trafficcontrol devices); the location of obstructions (e.g., roadwork,accidents, etc.); data indicative of events (e.g., scheduled concerts,parades, etc.); and/or any other map data that provides information thatassists the autonomous vehicle 105 in comprehending and perceiving itssurrounding environment and its relationship thereto. In someimplementations, the vehicle computing system 100 can determine avehicle route for the autonomous vehicle 105 based at least in part onthe map data 145.

The autonomous vehicle 105 can include a positioning system 150. Thepositioning system 150 can determine a current position of theautonomous vehicle 105. The positioning system 150 can be any device orcircuitry for analyzing the position of the autonomous vehicle 105. Forexample, the positioning system 150 can determine position by using oneor more of inertial sensors (e.g., inertial measurement unit(s), etc.),a satellite positioning system, based on IP address, by usingtriangulation and/or proximity to network access points or other networkcomponents (e.g., cellular towers, WiFi access points, etc.) and/orother suitable techniques. The position of the autonomous vehicle 105can be used by various systems of the vehicle computing system 100and/or provided to a remote computing system. For example, the map data145 can provide the autonomous vehicle 105 relative positions of theelements of a surrounding environment of the autonomous vehicle 105. Theautonomous vehicle 105 can identify its position within the surroundingenvironment (e.g., across six axes, etc.) based at least in part on themap data. For example, the vehicle computing system 100 can process thesensor data 140 (e.g., LIDAR data, camera data, etc.) to match it to amap of the surrounding environment to get an understanding of thevehicle's position within that environment.

The autonomy computing system 130 can include a perception system 155, aprediction system 160, a motion planning system 165, and/or othersystems that cooperate to perceive the surrounding environment of theautonomous vehicle 105 and determine a motion plan for controlling themotion of the autonomous vehicle 105 accordingly. For example, theautonomy computing system 130 can obtain the sensor data 140 from thevehicle sensor(s) 125, process the sensor data 140 (and/or other data)to perceive its surrounding environment, predict the motion of objectswithin the surrounding environment, and generate an appropriate motionplan through such surrounding environment. The autonomy computing system130 can communicate with the one or more vehicle control systems 135 tooperate the autonomous vehicle 105 according to the motion plan.

The vehicle computing system 100 (e.g., the autonomy computing system130) can identify one or more objects that are proximate to theautonomous vehicle 105 based at least in part on the sensor data 140and/or the map data 145. For example, the vehicle computing system 100(e.g., the perception system 155) can process the sensor data 140, themap data 145, etc. to obtain perception data 170. The vehicle computingsystem 100 can generate perception data 170 that is indicative of one ormore states (e.g., current and/or past state(s)) of a plurality ofobjects that are within a surrounding environment of the autonomousvehicle 105. For example, the perception data 170 for each object candescribe (e.g., for a given time, time period) an estimate of theobject's: current and/or past location (also referred to as position);current and/or past speed/velocity; current and/or past acceleration;current and/or past heading; current and/or past orientation;size/footprint (e.g., as represented by a bounding shape); class (e.g.,pedestrian class vs. vehicle class vs. bicycle class), the uncertaintiesassociated therewith, and/or other state information. The perceptionsystem 155 can provide the perception data 170 to the prediction system160 (and/or the motion planning system 165).

The prediction system 160 can be configured to predict a motion of theobject(s) within the surrounding environment of the autonomous vehicle105. For instance, the prediction system 160 can generate predictiondata 175 associated with such object(s). The prediction data 175 can beindicative of one or more predicted future locations of each respectiveobject. For example, the prediction system 160 can determine a predictedmotion trajectory along which a respective object is predicted to travelover time. A predicted motion trajectory can be indicative of a paththat the object is predicted to traverse and an associated timing withwhich the object is predicted to travel along the path. The predictedpath can include and/or be made up of a plurality of way points. In someimplementations, the prediction data 175 can be indicative of the speedand/or acceleration at which the respective object is predicted totravel along its associated predicted motion trajectory. The predictionsystem 160 can output the prediction data 175 (e.g., indicative of oneor more of the predicted motion trajectories) to the motion planningsystem 165.

The vehicle computing system 100 (e.g., the motion planning system 165)can determine a motion plan 180 for the autonomous vehicle 105 based atleast in part on the perception data 170, the prediction data 175,and/or other data. A motion plan 180 can include vehicle actions (e.g.,planned vehicle trajectories, speed(s), acceleration(s), other actions,etc.) with respect to one or more of the objects within the surroundingenvironment of the autonomous vehicle 105 as well as the objects'predicted movements. For instance, the motion planning system 165 canimplement an optimization algorithm, model, etc. that considers costdata associated with a vehicle action as well as other objectivefunctions (e.g., cost functions based on speed limits, traffic lights,etc.), if any, to determine optimized variables that make up the motionplan 180. The motion planning system 165 can determine that theautonomous vehicle 105 can perform a certain action (e.g., pass anobject, etc.) without increasing the potential risk to the autonomousvehicle 105 and/or violating any traffic laws (e.g., speed limits, laneboundaries, signage, etc.). For instance, the motion planning system 165can evaluate one or more of the predicted motion trajectories of one ormore objects during its cost data analysis as it determines an optimizedvehicle trajectory through the surrounding environment. The motionplanning system 165 can generate cost data associated with suchtrajectories. In some implementations, one or more of the predictedmotion trajectories may not ultimately change the motion of theautonomous vehicle 105 (e.g., due to an overriding factor). In someimplementations, the motion plan 180 may define the vehicle's motionsuch that the autonomous vehicle 105 avoids the object(s), reduces speedto give more leeway to one or more of the object(s), proceedscautiously, performs a stopping action, etc.

The motion planning system 165 can be configured to continuously updatethe vehicle's motion plan 180 and a corresponding planned vehicle motiontrajectory. For example, in some implementations, the motion planningsystem 165 can generate new motion plan(s) for the autonomous vehicle105 (e.g., multiple times per second). Each new motion plan can describea motion of the autonomous vehicle 105 over the next planning period(e.g., next several seconds). Moreover, a new motion plan may include anew planned vehicle motion trajectory. Thus, in some implementations,the motion planning system 165 can continuously operate to revise orotherwise generate a short-term motion plan based on the currentlyavailable data. Once the optimization planner has identified the optimalmotion plan (or some other iterative break occurs), the optimal motionplan (and the planned motion trajectory) can be selected and executed bythe autonomous vehicle 105.

The vehicle computing system 100 can cause the autonomous vehicle 105 toinitiate a motion control in accordance with at least a portion of themotion plan 180. A motion control can be an operation, action, etc. thatis associated with controlling the motion of the vehicle. For instance,the motion plan 180 can be provided to the vehicle control system(s) 135of the autonomous vehicle 105. The vehicle control system(s) 135 can beassociated with a vehicle controller (e.g., including a vehicleinterface) that is configured to implement the motion plan 180. Thevehicle controller can, for example, translate the motion plan intoinstructions for the appropriate vehicle control component (e.g.,acceleration control, brake control, steering control, etc.). By way ofexample, the vehicle controller can translate a determined motion plan180 into instructions to adjust the steering of the autonomous vehicle105 “X” degrees, apply a certain magnitude of braking force, etc. Thevehicle controller (e.g., the vehicle interface) can help facilitate theresponsible vehicle control (e.g., braking control system, steeringcontrol system, acceleration control system, etc.) to execute theinstructions and implement the motion plan 180 (e.g., by sending controlsignal(s), making the translated plan available, etc.). This can allowthe autonomous vehicle 105 to autonomously travel within the vehicle'ssurrounding environment.

The vehicle computing system 100 can a vigilance management system 185.The vigilance management system 185 can provide one or more vigilanceprompts, as described in greater detail herein, to manage a vehicleoperator's vigilance. In some implementations, the vigilance managementsystem 185 can be configured to operate in conjunction with the vehicleautonomy system 130. For example, the vigilance management system 185can send data to and receive data from the vehicle autonomy system 130.In some implementations, the vigilance management system 185 can beincluded in or otherwise a part of a vehicle autonomy system 130. Thevigilance management system 185 can include software and hardwareconfigured to provide the functionality described herein. In someimplementations, the v vigilance management system 185 can beimplemented as a subsystem of a vehicle computing system 100. Examplevigilance management system 185 configurations according to exampleaspects of the present disclosure are discussed in greater detail withrespect to FIGS. 3 and 4.

FIG. 2 depicts an example vehicle operator's perspective of an insidecab portion 200 of an autonomous vehicle according to example aspects ofthe present disclosure. The inside cab portion 200 can be, for example,an inside cab portion of an autonomous vehicle 105, and can be used toprovide vigilance prompts to a vehicle operator and receive responsesfrom the vehicle operator in response to the vigilance prompts, asdisclosed herein. The inside cab portion 200 depicted in FIG. 2 is forillustrative purposes only. Other suitable inside cab portionconfigurations can similarly be used to provide vigilance prompts andreceive responses from the vehicle operator.

Inside cab portion 200 can include one or more seats 210 in which avehicle operator can sit while manually operating the vehicle. Forexample, manual vehicle controls such as a gear shifter 215, pedals 220and a steering wheel 225 can be accessible to a vehicle operator sittingin seat 210.

In some implementations, inside cab portion 200 can also include one ormore display screens 230, such as touch-sensitive display screens. Thedisplay screens 230 can provide information to the vehicle operator,such as navigation information (e.g., GPS navigation) and/or otherinformation. The inside cab portion can also include one or morespeakers 235 configured to audibly play sounds for the vehicle operator.For example, the one or more speakers 235 can be used to play music orplay communications from a remote emergency operator. In someimplementations, inside cab portion 200 can also include one or moremicrophones (not shown). For example, the vehicle operator can have atwo-way communication with a remote emergency operator by speaking intothe microphone, and receiving responses from the remote emergencyoperator via the one or more speakers 235.

In some implementations, steering wheel 225 and/or components of aninside cab portion 200 can include various vehicle operator inputdevices. For example, the vehicle operator input devices can allow for avehicle operator to perform particular interactions with the vehicle.

For example, in some implementations, one or more shifter paddles 240can be included on or otherwise incorporated into a steering wheel 225.For example, as shown in FIG. 2, a first shifter paddle 240L ispositioned on a left side of steering wheel 225, and a second shifterpaddle 240R is positioned on a right side of steering wheel 225. Eachshifter paddle can be configured to manually control shifting betweengears. For example, a first shifter paddle 240L can be used to downshifta gear, and a second shifter paddle 240R can be used to upshift. Asdepicted in FIG. 2, in some implementations, shifter paddles 240L and240R can be configured to be accessible without a vehicle operatorremoving his/her hands from the steering wheel 225. In someimplementations, both the first and second shifter paddles 240L and 240Rcan be positioned on a single side of a steering wheel 225, on a frontportion of a steering wheel 225, proximate to a steering wheel 225 (suchas on a steering column assembly), or in any other suitableconfiguration.

In some implementations, a steering wheel 225 can include a windshieldwiper control arm 245. Windshield wiper control on 245 can be configuredto, for example, initiate control of a windshield wiper system in orderto remove rain and/or debris from a windshield of the vehicle. In someimplementations, windshield wiper control arm 245 can include controlsconfigured to, for example, select a windshield wiper actuationfrequency (e.g., a continuous or intermittent mode), spray thewindshield with windshield wiper fluid, cause the windshield wipers towipe the windshield (e.g., 1-3 quick wipes in succession), or otherwindshield wiper control actions.

In some implementations, steering wheel 225 can also include a turnindicator 250. For example, turn indicator 250 can be used to activate aleft turn signal, a right turn signal, and in some implementations, ahazard signal wherein both the left and right turn signals areactivated.

In some implementations, steering wheel 225 can include a purpose-builtbutton 255 for example, purpose-built button 255 can be an aftermarketbutton incorporated into a steering wheel 225. Purpose-built button 255can be used, for example, to receive a vehicle operator response inresponse to a vigilance prompt for the vehicle operator to push thepurpose-built button 255, as disclosed herein.

While several vehicle operator input devices 240, 245, 250, and 255 aredepicted in FIG. 2 as being included on or otherwise incorporated into asteering wheel 225, it should be noted that some or all of these vehicleoperator input devices may be located at other locations within aninside cab portion 200 of an autonomous vehicle. Further, other vehicleoperator input devices can similarly be included in an inside cabportion 200, such as included on or otherwise incorporated into thesteering wheel 225 or at other locations.

According to example aspects of the present disclosure, the one or moredisplay screens 230 and/or the one or more speakers 235 can be used toprovide vigilance prompts to a vehicle operator. For example, avigilance prompt can be a prompt for a vehicle operator to perform aparticular interaction with the autonomous vehicle. For example, avigilance prompt can be an uncomplicated tasks which is sufficient toengage a vehicle operator without compromising the vehicle operator'sability to assume manual control of the autonomous vehicle.

As examples, in some implementations, the vigilance prompts can bevisual cues displayed on a display screen of the autonomous vehicle. Forexample, a visual cue can be a color change (e.g., from a first color toa second color), an icon display depicting a particular interaction(e.g., a static icon or an animation), a text command, or other visualcue. For example, a vehicle operator can be informed that he or she isto touch the display screen 230 when the display screen changes color.Similarly, when a text command is displayed on the display screen 230,the vehicle operator can be prompted to perform the particularinteraction displayed. For example, as shown in FIG. 2, the text “LEFTSHIFTER PADDLE” is displayed on the display screen 230, which can be avigilance prompt for the vehicle operator to press the left shifterpaddle 240L. Various icons, animations, and other visual cues can belikewise used as vigilance prompts to prompt the vehicle operator toperform a particular interaction with the autonomous vehicle.

Similarly, in some implementations, one or more vigilance prompts can beprovided audio cues by the one or more speakers 235. For example, thespeakers 235 can be used to provide a command for the vehicle operatorto perform a particular interaction with the vehicle that the vehicleoperator hears over the one or more speakers 235. For example, thevehicle operator can be provided an audio cue, via the one or morespeakers 235 to, for example, “ACTIVATE LEFT TURN SIGNAL” or performsome other particular interaction.

In some implementations, the vigilance prompt can include both a visualcue and an audio cue. For example, a text command such as “LEFT SHIFTERPADDLE” can be displayed on a display screen 230 concurrently with anaudio cue with the same command.

In response, to receiving a vigilance prompt, the vehicle operator canprovide a response. For example, in response to “LEFT SHIFTER PADDLE”being displayed on the display screen 230, the vehicle operator canpress the left shifter paddle 240L. Similarly, in response to an audiocue to “ACTIVATE LEFT TURN SIGNAL” the vehicle operator can activate theturn indicator 250 to turn on the left turn signal.

In some implementations, the vehicle operator can provide a verbalresponse to a vigilance prompt. For example, the vehicle operator couldbe asked a question, such as by text displayed on a display screen 230or as sound played through the one or more speakers 235. In response,the vehicle operator can provide a verbal response, such as via one ormore microphones (not shown) configured to receive audio signals withinan inside cab portion 200 of an autonomous vehicle.

According to example aspects of the present disclosure, a vehicleoperator can be provided a variety of vigilance prompts, such as atregular or random intervals, in order to engage and stimulate thevehicle operator in order to improve the vehicle operator's alertness.Further, the variety of vigilance prompts provided herein can help toprevent and overcome any tendency of a vehicle operator to becomingconditioned to respond to a vigilance prompt without improving thevehicle operator's alertness, such as can happen when only a single typeof prompt is provided.

Referring now to FIG. 3, an example vigilance management system 300according to example aspects of the present disclosure is depicted. Thevigilance management system 300 can be implemented by one or moreprocessors and one or more memory devices, such as, for example, avehicle computing system 100. The vigilance management system 300 can beused to manage a vehicle operator's vigilance level by periodicallytesting the vehicle operator via one or more vigilance prompts.

As shown, a vigilance prompt and interaction 305 can be provided to avehicle operator of an autonomous vehicle. The vigilance prompt can be,for example, a visual cue displayed on a display screen of theautonomous vehicle or an audio cue provided by a speaker of theautonomous vehicle, as disclosed herein. In response, the vehicleoperator can provide a response 310. For example, the vigilance promptcan prompt the vehicle operator to perform a particular user interactionwith the autonomous vehicle, and in response, the vehicle operator canperform the particular interaction, as disclosed herein.

A test output 315 can then be provided to a vigilancemanagement/normalization 320 based at least in part on the vehicleoperator's response 310 to the vigilance prompt and interaction 305. Forexample, the test output 315 can describe how quickly the vehicleoperator responded (e.g., a response time), an accuracy of the response310 (e.g., whether the vehicle operator performed the correctinteraction), and/or other information, such as information about theparticular vigilance prompt and interaction 305 provided to the vehicleoperator.

The vigilance management/normalization 320 can then determine a vehicleoperator management action based at least in part on the test output315. For example, the vigilance management/normalization 320 candetermine a vigilance level based at least in part on the test output315. In some implementations, the vigilance level can be determinedbased on an accuracy of the test output 315, such as whether the vehicleoperator performed the particular interaction in the vigilance promptand interaction 310 correctly (e.g., pushed the correct shifter paddle,etc.). In some implementations, the vigilance level can be determinedbased on a response time of the test output 315, such as the time ittook for the vehicle operator to provide the response 310 following thevigilance prompt and interaction 305. For example, the vigilancemanagement/normalization 320 can reference lookup tables for particularvigilance prompts, such as acceptable response times and/or acceptableaccuracy rates for the particular vigilance prompts. In someimplementations, the vigilance level can be expressed as a numericalvalue (e.g., on a scale), a percentage, or other suitable format. Insome implementations, the vigilance level can be expressed as category(e.g. pass/fail, high/medium/low, responsive/nonresponsive, etc.). Insome implementations, the vigilance level can be determined based on aplurality of vehicle operator responses 310, such as a plurality of themost recently received responses 310 (e.g., a mean, median, rollingwindow, etc.). In this way, the vigilance level determined by vigilancemanagement/normalization 320 can be indicative of the overall alertnessand vigilance of the vehicle operator, both historically as well as at aparticular time. In some implementations, each vehicle operator can havean associated profile, and respective vehicle operator vigilance levelscan be tracked over time, such as over a plurality of autonomous drivingsessions.

In some implementations, the vigilance management/normalization 320 candetermine a vehicle operator management action based at least in part ontest output 315 and/or the vigilance level. For example, in someimplementations, the vehicle operator management action can be toprovide one or more subsequent vigilance prompts to the vehicleoperator.

For example, the vigilance management/normalization 320 can use a testgenerator 325 to generate the one or more subsequent vigilance prompts.For example, the test generator 325 can access a test database 330,which can include a variety of vigilance prompts. The test generator 325can determine how often the one or more subsequent vigilance prompts canbe provided, as well as a type of vigilance prompt to provide. Forexample, the test generator can determine the one or more subsequentvigilance prompts and one or more intervals at which they are providedbased at least in part on the test output 315, the vigilance level, arandom selection, a pattern, a vehicle operator profile, and/or otherfactors. For example, if a vehicle operator vigilance level is low, thetest generator 325 can determine a time delay based at least in part onthe test output 315 and/or the vehicle operator vigilance level. Forexample, for a low vigilance level, the time delay for a subsequentvigilance prompt can be a short time delay, such as a five minute delayas compared to a 10 minute delay for a middle vigilance level or a 15minute delay for a high vigilance level. Stated differently, thevigilance management system 300 can use the vehicle operator vigilancelevel to determine (e.g., adjust) the frequency of one or moresubsequent vigilance prompts. By providing subsequent vigilance promptsat a higher frequency, the vehicle operator's vigilance can be increasedby stimulating the vehicle operator more frequently. Similarly, forvehicle operators with a high vigilance level, the frequency ofsubsequent vigilance prompts can, in some instances, be reduced, as thevehicle operator or may not require additional stimulation to maintainhis/her vigilance.

In some implementations, the vehicle operator management action can beto determine (e.g., adjust) the type of one or more subsequent vigilanceprompts. For example, certain vigilance prompts and their associatedparticular interactions may be better at stimulating and engaging thevehicle operator, and therefore more effective in maintaining thevehicle operator's vigilance. For example, in some implementations,analysis of a particular vehicle operator's profile may indicate thattouching a display screen in response to a color change is moreeffective at increasing the vehicle operator vigilance level thanpressing a button in response to an icon display. Accordingly, if thevehicle operator vigilance level is low, the test generator 325 canincrease the frequency of the display screen touch vigilance promptrelative to the button press vigilance prompt. Similarly, for somevehicle operators, audio cues may be more effective in maintaining avehicle operator vigilance level than visual cues, or vice-versa.Accordingly, if a vehicle operator vigilance level is low, the testgenerator 325 can provide audio cues rather than visual cues, orvice-versa. In this way, a vigilance management system 300 can determinea vigilance prompt based at least in part on the test output 315 and/ora vehicle operator vigilance level.

Similarly, in some implementations, the type of vigilance prompt caninclude both a visual cue and an audio cue. For example, if a vehicleoperator vigilance level is low, such as following several consecutivevigilance prompts, the test generator 325 can escalate the type ofvigilance prompt by concurrently providing both an audio cue and avisual cue with each vigilance prompt. By providing both a visual cueand an audio cue, the vehicle operator may be stimulated, and thereforehave improved vigilance.

In some implementations, the test generator 325 can include a cognitiveloading component 335 in a vigilance prompt. The cognitive loadingcomponent 335 can be, for example, a cognitive load aspect of avigilance prompt which requires the vehicle operator to perform a moreor less complicated task. For example, the variety of vigilance promptsand the amount of time between the vigilance prompts can be increased orreduced to increase or decrease the cognitive loading component 335 ofone or more vigilance prompts. Similarly, more complex or less complexvigilance prompts can be provided, such as a sequence of interactions(e.g., a first action followed by a second action), etc. In someimplementations, such as in a testing environment, the cognitive loadingcomponent 335 can be used to normalize vehicle operator response times.For example, reduced cognitive loading components 335 can improvevehicle operator response times, while increased cognitive loadingcomponents 335 can slow vehicle operator response times. In this way,individual vehicle operator performance can be normalized, which canhelp prevent testing results from being skewed by hyper-vigilant vehicleoperators.

The test generator 325 can then provide the vigilance prompts to theprompt display and interaction 305. For example, in someimplementations, the test generator 325 can preload a vigilance promptin the prompt display and interaction 305 by spooling 340 the prompt(also referred to as a test). The test generator can then provide atrigger 345 to the test display and interaction 305 at an appropriatetime for the vigilance prompt to be provided to the vehicle operator.For example, the spooling 340 can preload a vigilance prompt, which canthen be provided when the trigger 345 is received.

In some implementations, the vehicle management system 300 can implementother vehicle operator management actions. For example, in someimplementations, the vehicle operator can be provided feedback on thevehicle operator vigilance level. For example, the vehicle managementsystem 300 can display or audibly play an indicator of the vehicleoperator vigilance level (e.g., a numerical value, score, percentage,accuracy, response time, or category, such as pass/fail, high/low,etc.). In some implementations, a vehicle operator vigilance level for aparticular vigilance prompt can be provided, such as an indication ofthe accuracy or response time following a response to a particularvigilance prompt. In some implementations, the vehicle operatorvigilance level can track a vehicle operator's vigilance over aplurality of vigilance prompts, which can indicate whether a vehicleoperator's vigilance is improving or decreasing. By providing thevehicle operator with feedback on the vehicle operator vigilance level,the vehicle operator's interest and motivation to perform well may causethe vehicle operator to become more vigilant.

In some implementations, the vehicle operator management action caninclude operating the autonomous vehicle to a safe state in whichautonomous operation is disabled. For example, if a vehicle operator'svigilance level is too low, such as following a failure to respond toone or more consecutive vigilance prompts, the vigilance managementsystem 300 can implement a motion plan to operate the vehicle to a safestate, such as navigating the autonomous vehicle to a stop in a parkinglot, and autonomous operation can be disabled. For example, the vehicleoperator can be prevented from causing the autonomous vehicle to enteran autonomous operation mode until a reset has occurred, such as afterthe vehicle operator has been contacted by an external vehicle operatormanagement system, as disclosed herein. Further, the vehicle operatormay only be able to operate the autonomous vehicle in a manual operatingmode until the reset has occurred.

Similarly, in some implementations, the autonomous vehicle may beconfigured to operate autonomously for one or more discrete portions ofa motion plan, and at the end of each portion, the vehicle operator maybe required to engage or reengage autonomous operation before theautonomous vehicle proceeds to initiate travel in accordance with themotion plan. If the vehicle operator vigilance level is too low, in someimplementations, the vehicle operator may be prevented from causing theautonomous vehicle to enter an autonomous operating mode. In someimplementations, the autonomous vehicle can be disengaged from anautonomous operating mode in response to a low vehicle operatorvigilance level.

In some implementations, a vehicle operator management action caninclude contacting an external vehicle operator management system. Forexample, an autonomous vehicle can include a communication systemconfigured to communicate with an external vehicle operator managementsystem. The external vehicle operator management system can be remotefrom the vehicle, and can provide various services such as securityservices, emergency services, roadside assistance, remote diagnostics,and/or communication services, such as with an emergency operator. Insome implementations, the external vehicle operator management systemcan assist the vehicle operator and/or remotely diagnose one or moreissues with the autonomous vehicle. For example, if a vehicle operatorvigilance level is low, a remote operator can ask the vehicle operatorif there is anything in particular affecting the vehicle operator'sability to respond to vigilance prompts. For example, external weatherconditions or traffic conditions may impair a vehicle operator's abilityto respond to vigilance prompts due to the vehicle operator paying closeattention to the conditions in which the autonomous vehicle isoperating. Similarly, a vehicle operator input device, such as amicrophone, may be malfunctioning, and the external vehicle operatormanagement system may be able to remotely diagnose such malfunctions. Insuch situations, the external vehicle operator management system candetermine that the vehicle operator's vigilance level should not preventthe vehicle operator from causing the vehicle to enter or continueoperating in an autonomous operating mode. Similarly, the externalvehicle operator management system can reset or clear any disablement ofautonomous operation, such as when the autonomous vehicle has beenoperated to a safe state or the vehicle operator has been prevented fromcausing the vehicle to enter an autonomous operating mode.

In some implementations, the vehicle operator management action caninclude logging the vehicle operator vigilance level, such as for lateranalysis. For example, the vigilance management system 300 can locallystore data indicative of the vehicle operator vigilance level, such asindividual responses, response accuracy levels, response times, or otherdata related to or indicative of the vehicle operator vigilance level.The vigilance management system 300 can further be configured tocommunicate the logged data, such as over a communications network, toan external computing system. In some implementations, analysis can beperformed on the logged data in order to identify ways to maintainand/or improve a vehicle operator vigilance level. For example, machinelearned models can be used to identify effective vigilance prompts, bothon an individual level (e.g., a particular vehicle operator profile) orover a plurality of vehicle operators (e.g., aggregated vehicle operatorprofiles).

Referring now to FIG. 4, an example vigilance management system 400according to example aspects of the present disclosure is depicted. Thevigilance management system 400 can be implemented by one or moreprocessors and one or more memory devices, such as, for example, avehicle computing system 100. The vigilance management system 400 can beused to manage a vehicle operator's vigilance level by periodicallytesting the vehicle operator via one or more vigilance prompts.

Similar to vigilance management system 300, a vigilance prompt 402 canbe provided and a response 404 can be received. For example, a promptdisplay and interaction 406 can be a visual and/or audible prompt 408,such as a vigilance prompt 402 displayed on a display screen 410. Insome implementations, the prompt display and interaction 406 can alsoreceive a response 404 via the display screen 410. For example, thedisplay screen 410 can be a touch-sensitive display screen, and avehicle operator can provide a response 404 via the display screen 410in response to the visual/audible prompt 408. In response to receiving aresponse 404, the prompt display and interaction 406 can providetimestamped test data 414. Further, the prompt display and interaction406 can provide a test readiness 412. Test readiness 412 can beindicative of, for example, whether a vehicle operator is ready to beprovided a vigilance prompt 402, such as how much time has elapsed sincethe vehicle operator was last provided a vigilance prompt 402 and/or thetime the last vigilance prompt 402 was provided.

The response 404 can also be received from the vehicle operator viavarious vehicle operator input devices. For example, in someimplementations, the response 404 can be received via a peripheral input416. For example, a button 418 (e.g., a purpose-built button affixed toa steering wheel or other location) can receive the response 404, andprovide the response 404 to a vehicle interface module 426. In someimplementations, the response 404 can be received via a steering wheelinput 420. For example, repurposed shifter paddles 422 can receive theresponse 404, and can provide the response to the vehicle interfacemodule 426. In some implementations, the response 404 can be analternate input received via a vehicle platform 424, such as turnindicator actuation, windshield wiper actuation, voice command receivedvia a microphone, or other interaction with a vehicle platform 424.

Vehicle interface module 426 can be, for example, a vehicle controllerand associated network (e.g., harnesses, connections, wires, etc.) toallow various components of a vehicle to interface (e.g., communicate),such as to control the motion of the autonomous vehicle. Variouscommands and data can be sent and received by the vehicle interfacemodule 426. In some implementations, vehicle interface module 426 canprovide a visual and/or audible prompt 430, such as over one or morespeakers or display screens of the vehicle, etc. Vehicle interfacemodule 426 can receive the responses 404 from the various vehicleoperator input devices, as well as other inputs from the vehicleplatform 428. Responsive to receiving a response 404, vehicle interfacemodule 426 can provide timestamped test data 414 to vigilance management438. Further, vehicle interface module 426 can provide a test readiness432. As with test readiness 412, test readiness 432 can be indicative ofwhether a vehicle operator is ready to be provided a vigilance prompt402, such as how much time has elapsed since the vehicle operator waslast provided a vigilance prompt 402 and/or the time the last vigilanceprompt 402 was provided.

Vigilance management 438 can receive the timestamped test data 414 fromthe vehicle interface module 426 and the prompt display and interaction406. Vigilance management 438 can further receive test readiness 436,which can include test readiness 432, test readiness 412, and/or a testreadiness from autonomy 434. Similar to test readiness 412 and 432, testreadiness 436 can be indicative of, for example, whether a vehicleoperator is ready to be provided a vigilance prompt 402, such as howmuch time has elapsed since the vehicle operator was last provided avigilance prompt 402 and/or the time the last vigilance prompt 402 wasprovided. Test readiness 436 can also include data from autonomy 434,such as data indicative of a motion plan for the autonomous vehicle. Forexample, in some implementations, vigilance management 438 can determinefrom test readiness 436 whether any pending lane changes or turns withina testing period are planned in the motion plan. The testing period canbe, for example, the time needed to provide a vigilance test, includingproviding a vigilance prompt 402 and receiving a response 404. Further,in some implementations, vigilance management 438 can determine whetherthe autonomous vehicle is operating within an authorized area, such asan area in which providing vigilance prompts 402 to a vehicle operatorare allowed. In some implementations, if the autonomous vehicle isoperating within an authorized area without any pending turns or lanechanges within a testing period, vigilance management 438 can determinethat the vehicle operator is ready to be provided a vigilance prompt402. In some implementations, if the vehicle operator has not beenprovided a previous vigilance prompt 402 within a threshold time period,vigilance management 438 can determine that the vehicle operator isready to be provided a vigilance prompt 402.

Vigilance management 438 can use a test generator 440 to generate theone or more subsequent vigilance prompts 402. For example, the testgenerator 440 can access stored tests 442 (e.g., a test database), whichcan include a variety of vigilance prompts 402. The test generator 440can determine how often the one or more subsequent vigilance prompts 402can be provided, as well as a type of vigilance prompt 402 to provide.For example, the test generator 440 can determine the one or moresubsequent vigilance prompts 402 and one or more intervals at which theyare provided based at least in part on a vigilance level, a randomselection, a pattern, a vehicle operator profile, and/or other factors,as described herein. Stated differently, test generator 440 candetermine and/or adjust a frequency of one or more subsequent vigilanceprompts 402. Test generator 440 can further determine and/or adjust thetype of one or more subsequent vigilance prompts 402, such as whichvigilance prompt 402 is provided at a particular time. In someimplementations, a vigilance prompt 402 can also include a cognitiveloading component, as described herein (not shown).

The test generator 440 can then provide the vigilance prompts 402 to theprompt display and interaction 406 and the vehicle interface module 426.For example, in some implementations, the test generator 440 can preloada vigilance prompt 402 in the prompt display and interaction 406 byspooling 444 the vigilance prompt 402. The test generator 440 can thenprovide a trigger 446 to the test display and interaction 406 anappropriate time for the vigilance prompt 402 to be provided to thevehicle operator. For example, the spooling 444 can preload a vigilanceprompt 402, which can then be provided when the trigger 446 is received.Additional spooling path 448 and additional trigger path 450 canlikewise be used by test generator 440 to spool and trigger,respectively, a vigilance prompt 402 via a vehicle interface module 426.

Vigilance management system 400 can further be configured to provideadditional vehicle operator management actions, such as providing avehicle operator with vigilance level feedback, operating and autonomousvehicle to a safe state in which autonomous operation is disabled,preventing the vehicle operator from causing the autonomous vehicle toenter an autonomous operating mode, contacting an external vehicleoperator management system, or logging the vehicle operator vigilancelevel, as disclosed herein.

Referring now to FIG. 5, an example vehicle operator vigilancemanagement process 500 according to example aspects of the presentdisclosure is depicted. The vehicle operator vigilance managementprocess 500 can be implemented by one or more processors and one or morememory devices, such as, for example, a vehicle computing system 100and/or a vigilance management system 300/400 operating thereon. Thevehicle operator vigilance management process 500 can be used, forexample, to manage a vehicle operator's vigilance level by periodicallytesting the vehicle operator via one or more vigilance prompts.

As shown, driver input 502 (also referred to as vehicle operatorresponse 502) can be received. For example, a vehicle operator canprovide a response to a vigilance prompt via a vehicle operator inputdevice.

At 504, it can be determined whether a test was actually in process(e.g., whether a vigilance prompt had been provided). If not, thevehicle operator can be provided a warning 506, such as an audiblewarning 506. Further, the vehicle operator response 502 can be logged at508. For example, data indicative of the vehicle operator response 502can be stored locally on a memory of an onboard computing system orcommunicated to an external computing system. A timestamp 510 cansimilarly be logged at 508, which can indicate a time at which thevehicle operator response 502 was received.

Whether or not a test was in progress at 504, the vehicle operatorresponse 502, test data 512, and a timestamp 514 can be used to updatevehicle operator vigilance metrics 516. For example, in someimplementations, the vehicle operator vigilance metrics 516 can be orotherwise include a vehicle operator vigilance level, as disclosedherein. The updated vehicle operator vigilance metrics 516 can bedetermined based at least in part on, for example, an accuracy of thevehicle operator response 502 and/or a response time of the vehicleoperator response 502. The updated vehicle operator vigilance metrics516 can also be logged at 508. For example, data indicative of updatedvehicle operator metrics 516 can be stored locally on a memory of anonboard computing system or communicated to an external computingsystem. The timestamp 514 and test data 512 can also be logged at 508.

At 518, the updated vehicle operator vigilance metrics 516 can becompared to one or more driver performance thresholds 520 (also referredto as vehicle operator performance thresholds 520). For example, it canbe determined whether the updated vehicle operator vigilance metrics 516are acceptable. The vehicle operator performance thresholds 520 can be,for example, vehicle operator performance thresholds 520 determined forspecific drivers, operational areas (e.g., geographical areas), or othervehicle operator performance thresholds 520. In various implementations,heuristics, ranges, thresholds, fuzzy logic, and/or machine-learnedmodels can be used to determine whether a vehicle operator's vigilancemetrics 516 are acceptable. In some implementations, the vehicleoperator performance thresholds 520 can be static thresholds (e.g.,thresholds which do not change), while in other implementations, thevehicle operator performance thresholds 520 can be dynamic thresholds(e.g., thresholds which can be varied depending on various factors).

If at 518 it is determined that the vehicle operator vigilance metrics516 are not acceptable, at 522, re-entry (or entry) into autonomousoperation can be disabled. For example, in some implementations, theautonomous vehicle may be configured to operate autonomously for one ormore discrete portions of a motion plan, and at the end of each portion,the vehicle operator may be required to cause the autonomous vehicle toengage or reengage autonomous operation before the autonomous vehicleproceeds to initiate travel in accordance with the motion plan. In someimplementations, if the vehicle operator vigilance metrics 516 are notacceptable, the autonomous vehicle can be disengaged from an autonomousoperating mode, such as by alerting the vehicle operator to aforthcoming disengagement of the autonomous operator mode and navigatingthe autonomous vehicle to a safe state prior to disengaging autonomousoperation. If the updated vehicle operator vigilance metrics 516 are toolow (e.g., not acceptable), in some implementations, the vehicleoperator may be prevented from causing the autonomous vehicle to enter(or re-enter) an autonomous operating mode. In some implementations, thevehicle operator may only be able to manually control the autonomousvehicle. In some implementations, the autonomous vehicle can be operatedto a safe state and disabled, as disclosed herein. At 524, the vehicleoperator can be provided a visual notice that entry (or re-entry) intoautonomous operation has been disabled. At 526, the vehicle operator canbe provided an audible notice at entry (or re-entry) into autonomousoperation has been disabled. The visual notice 524 and/or audible notice526 can also be logged at 508, which can include a timestamp 510 of whenthe notices 524/526 were provided.

If at 518 the vehicle operator vigilance metrics 516 are acceptable, at528, it can be determined whether the vehicle operator responded withthe correct input. If not, in some implementations, at 530, the varietyof subsequent vigilance prompts to be provided to the vehicle operatorcan be reduced. For example, the types of prompts which may be providedin subsequent tests can be limited to only certain vigilance prompts.Further, at 524 and/or 526, a visual notice and/or audible notice,respectively, can be provided to the vehicle operator indicating thatthe driver did not respond with the correct input. The visual notice 524and/or audible notice 526 can also be logged at 508, which can include atimestamp 510 of when the notices 524/526 were provided.

If at 528 the vehicle operator responded with the correct input, at 532,test varieties can be restored. For example, the full variety ofpossible vigilance prompts can be used in subsequent tests. At 534, itcan be determined whether the vehicle operator responded within a timethreshold for the cued action (e.g., whether the vehicle operator'sresponse time was less than the time threshold). If not, at 536, in someimplementations, the interval between subsequent prompts can be reduced.For example, subsequent prompts can be provided at an increasedfrequency. Further, at 526, an audible notice can be provided to thevehicle operator indicating that the driver did not respond within thetime threshold. The audible notice 526 can also be logged at 508, whichcan include a timestamp 510 of when the audible notice 526 was provided.In some implementations, a visual notice 524 can similarly be providedand logged at 508.

If at 534 the vehicle operator responded within the time threshold forthe cued action, then at 538, test intervals and varieties can beincreased and/or restored. For example, the interval between tests(e.g., providing subsequent vigilance prompts and receiving subsequentresponses) can be increased back to a nominal time period. Theincreased/restored test interval 538 can be logged a 508, which caninclude a timestamp 510 of when the increased/restored test interval 538was implemented.

Referring now to FIG. 6, an example method (600) for determining avigilance level of a vehicle operator in an autonomous vehicle accordingto example aspects of the present disclosure is depicted. Although FIG.6 depicts steps performed in a particular order for purposes ofillustration and discussion, the methods of the present disclosure arenot limited to the particularly illustrated order or arrangement. Thevarious steps of method (600) can be omitted, rearranged, combined,and/or adapted in various ways without deviating from the scope of thepresent disclosure. The method (600) can be implemented by one or moreprocessors and one or more tangible, non-transitory computer-readablememory.

At (602), the method (600) can include determining a vehicle operator isready for a first vigilance prompt. For example, in someimplementations, determining that the vehicle operator is ready for afirst vigilance prompt can include determining that the autonomousvehicle is operating within an authorized area without any pending turnsor lane changes within a testing period. In some implementations,determining that the vehicle operator is ready to be provided the firstvigilance prompt can include determining that the vehicle operator hasnot been provided a previous vigilance prompt within a threshold timeperiod.

At (604), the method (600) can include determining a first vigilanceprompt from a plurality of vigilance prompts. For example, the firstvigilance prompt can be a visual cue, such as a visual cue displayed ona display screen of an autonomous vehicle and/or an audio cue played bya speaker of the autonomous vehicle. For example, visual cue can be acolor change, an icon display, or a text command, which can be displayedon the display screen. Each of the plurality of vigilance prompts can bedifferent from each other vigilance prompt, and can be a prompt for thevehicle operator to perform a particular interaction with the autonomousvehicle. For example, in some implementations, the particularinteraction can be a button push, a display screen touch, a shifterpaddle actuation, a turn indicator actuation, a windshield wiperactuation, or a verbal response.

At (606), the method (600) can include providing the first vigilanceprompt. For example, the first vigilance prompt can be displayed on thedisplay screen or audibly played for the vehicle operator by a speakerof the autonomous vehicle.

At (608), the method (600) can include receiving a first response to thefirst vigilance prompt. For example, the vehicle operator can performthe particular interaction prompted by the first vigilance prompt. Thefirst response can be received by a vehicle operator input device.

At (610), the method (600) can include determining a vehicle operatorvigilance level based at least in part on the first response. Forexample, in some implementations, the vehicle operator vigilance levelcan be determined based at least in part on an accuracy of the firstresponse or a response time of the first response. In someimplementations, the vigilance level can be expressed as a numericalvalue (e.g., on a scale), a percentage, or other suitable format. Insome implementations, the vigilance level can be expressed as category(e.g. pass/fail, high/medium/low, responsive/nonresponsive, etc.). Insome implementations, the vigilance level can be determined based on aplurality of vehicle operator responses, such as a plurality of the mostrecently received responses (e.g., a mean, median, rolling window,etc.). In some implementations, a plurality of vehicle operatorresponses can be weighted differently. For example, more recentlyprovided responses may be weighted more than older responses. In someimplementations, a vigilance level can include a historical vigilancelevel component, such as an average over a plurality of vigilanceprompts and responses, as well as a recent vigilance level component,such as one or more of the most recent prompts and responses. Thus, insome implementations, the vigilance level can be indicative of theoverall alertness and vigilance of the vehicle operator, bothhistorically as well as at a particular time. In some implementations,each vehicle operator can have an associated profile, and respectivevehicle operator vigilance levels can be tracked over time, such as overa plurality of autonomous driving sessions.

At (612), the method (600) can include determining a vehicle operatormanagement action based at least in part on the vigilance level. Forexample, in some implementations, the vehicle operator management actioncan include determining or adjusting a frequency of one or moresubsequent vigilance prompts, determining or adjusting a type of one ormore subsequent vigilance prompts, providing the vehicle operator withvigilance level feedback, operating the autonomous vehicle to a safestate in which autonomous operation is disabled, preventing the vehicleoperator from causing the autonomous vehicle to enter an autonomousoperating mode, contacting an external vehicle operator managementsystem, or logging the vehicle operator vigilance level. The vehicleoperator management action can then be implemented at (614).

Referring now to FIG. 7, an example method (700) according to exampleaspects of the present disclosure is depicted. Although FIG. 7 depictssteps performed in a particular order for purposes of illustration anddiscussion, the methods of the present disclosure are not limited to theparticularly illustrated order or arrangement. The various steps ofmethod (700) can be omitted, rearranged, combined, and/or adapted invarious ways without deviating from the scope of the present disclosure.Method (700) can be implemented in an autonomous vehicle configured tooperate in a non-public testing environment, such as on an isolatedtesting track, by one or more processors and one or more tangible,non-transitory computer-readable memory of the autonomous vehicle.

At (702), the method (700) can include providing a first vigilanceprompt. For example, the first vigilance prompt can be displayed on thedisplay screen or audibly played for the vehicle operator by a speakerof the autonomous vehicle. The first vigilance prompt can be a visualcue, such as a visual cue displayed on a display screen of an autonomousvehicle, and/or an audio cue played by a speaker of the autonomousvehicle. For example, a visual cue can be a color change, an icondisplay, or a text command, which can be displayed on the displayscreen. The first vigilance prompt can be a prompt for the vehicleoperator to perform a particular interaction with the autonomousvehicle. For example, in some implementations, the particularinteraction can be a button push, a display screen touch, a shifterpaddle actuation, a turn indicator actuation, a windshield wiperactuation, or a verbal response.

At (704), the method (700) can include receiving a first response to thefirst vigilance prompt. For example, the vehicle operator can performthe particular interaction prompted by the first vigilance prompt. Thefirst response can be received by a vehicle operator input device.

At (706), the method (700) can include determining a second vigilanceprompt comprising a cognitive loading component based at least in parton the first response. For example, the cognitive loading component canbe a cognitive load aspect of a vigilance prompt which requires thevehicle operator to perform a more or less complicated task. Forexample, rather than pushing a button, the vehicle operator may berequired to press a sequence of buttons, solve a mathematical questionand provide a verbal response, or perform some other cognitive task.Similarly, in some implementations, the second vigilance prompt can beselected from a variety of vigilance prompts, and the variety ofvigilance prompts can be narrowed or increased based at least in part onthe first response. Further, in some implementations, the amount of timebetween the first vigilance prompt and the second vigilance prompt canbe increased or decreased, such as from a nominal time, based at leastin part on the first response.

At (708), the method (700) can include providing the second vigilanceprompt. For example, the second vigilance prompt can be displayed on thedisplay screen or audibly played for the vehicle operator by a speakerof the autonomous vehicle.

FIG. 8 depicts an example system 800 according to example embodiments ofthe present disclosure. The example system 800 illustrated in FIG. 8 isprovided as an example only. The components, systems, connections,and/or other aspects illustrated in FIG. 8 are optional and are providedas examples of what is possible, but not required, to implement thepresent disclosure. The example system 800 can include a vehiclecomputing system 805 of a vehicle. The vehicle computing system 805 canrepresent/correspond to the vehicle computing system 100 describedherein. The example system 800 can include a remote computing system 850(e.g., that is remote from the vehicle computing system). The remotecomputing system 850 can represent/correspond to an external vehicleoperator management system described herein. The vehicle computingsystem 805 and the remote computing system 850 can be communicativelycoupled to one another over one or more network(s) 840.

The computing device(s) 810 of the vehicle computing system 805 caninclude processor(s) 815 and a memory 820. The one or more processors815 can be any suitable processing device (e.g., a processor core, amicroprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.)and can be one processor or a plurality of processors that areoperatively connected. The memory 820 can include one or morenon-transitory computer-readable storage media, such as RAM, ROM,EEPROM, EPROM, one or more memory devices, flash memory devices, dataregistrar, etc., and combinations thereof.

The memory 820 can store information that can be accessed by the one ormore processors 815. For instance, the memory 820 (e.g., one or morenon-transitory computer-readable storage mediums, memory devices)on-board the vehicle can include computer-readable instructions 825 thatcan be executed by the one or more processors 815. The instructions 825can be software written in any suitable programming language or can beimplemented in hardware. Additionally, or alternatively, theinstructions 825 can be executed in logically and/or virtually separatethreads on processor(s) 815.

For example, the memory 820 can store instructions 825 that whenexecuted by the one or more processors 815 cause the one or moreprocessors 815 (the vehicle computing system 805) to perform operationssuch as any of the operations and functions of the vehicle computingsystem 100 (or for which it is configured), one or more of theoperations and functions of the operations computing systems describedherein (or for which it is configured), one or more of the operationsand functions for determining vehicle operator vigilance levelsdescribed herein, one or more portions of methods 600 and 700 describedherein, and/or one or more of the other operations and functions of thecomputing systems or vehicle operator vigilance management systemsdescribed herein.

The memory 820 can store data 830 that can be obtained (e.g., acquired,received, retrieved, accessed, created, stored, etc.). The data 830 caninclude, for instance, sensor data, map data, vehicle state data,perception data, prediction data, motion planning data, vigilance leveldata, test databases (e.g., stored tests), timestamped test data,vigilance prompt data, and/or other data/information such as, forexample, that described herein. In some implementations, the computingdevice(s) 810 can obtain data from one or more memories that are remotefrom the vehicle computing system 805.

The computing device(s) 810 can also include a communication interface835 used to communicate with one or more other system(s) on-board avehicle and/or a remote computing device that is remote from the vehicle(e.g., of the system 850). The communication interface 835 can includeany circuits, components, software, etc. for communicating via one ormore networks (e.g., network(s) 840). The communication interface 835can include, for example, one or more of a communications controller,receiver, transceiver, transmitter, port, conductors, software and/orhardware for communicating data.

The remote computing system 850 can include one or more computingdevice(s) 855 that are remote from the vehicle computing system 805. Thecomputing device(s) 855 can include one or more processors 860 and amemory 865. The one or more processors 860 can be any suitableprocessing device (e.g., a processor core, a microprocessor, an ASIC, aFPGA, a controller, a microcontroller, etc.) and can be one processor ora plurality of processors that are operatively connected. The memory 865can include one or more tangible, non-transitory computer-readablestorage media, such as RAM, ROM, EEPROM, EPROM, one or more memorydevices, flash memory devices, data registrar, etc., and combinationsthereof.

The memory 865 can store information that can be accessed by the one ormore processors 860. For instance, the memory 865 (e.g., one or moretangible, non-transitory computer-readable storage media, one or morememory devices, etc.) can include computer-readable instructions 870that can be executed by the one or more processors 860. The instructions870 can be software written in any suitable programming language or canbe implemented in hardware. Additionally, or alternatively, theinstructions 870 can be executed in logically and/or virtually separatethreads on processor(s) 860.

For example, the memory 865 can store instructions 870 that whenexecuted by the one or more processors 860 cause the one or moreprocessors 860 to perform operations. For example, the operations caninclude operations to analyze vigilance test data (e.g, test outputdata, test databases, stored tests, cognitive load components), analyzevehicle operator vigilance levels, disable autonomous operation on anautonomous vehicle, and/or prevent a vehicle operator from causing anautonomous vehicle from entering or reentering an autonomous operatingmode.

The memory 865 can store data 875 that can be obtained. The data 875 caninclude, can include, for instance, sensor data, map data, vehicle statedata, perception data, prediction data, motion planning data, vigilancelevel data, timestamped test data, vigilance prompt data, and/or otherdata/information described herein.

The computing device(s) 855 can also include a communication interface880 used to communicate with one or more system(s) onboard a vehicleand/or another computing device that is remote from the system 850. Thecommunication interface 880 can include any circuits, components,software, etc. for communicating via one or more networks (e.g.,network(s) 840). The communication interface 880 can include, forexample, one or more of a communications controller, receiver,transceiver, transmitter, port, conductors, software and/or hardware forcommunicating data.

The network(s) 840 can be any type of network or combination of networksthat allows for communication between devices. In some embodiments, thenetwork(s) 840 can include one or more of a local area network, widearea network, the Internet, secure network, cellular network, meshnetwork, peer-to-peer communication link and/or some combination thereofand can include any number of wired or wireless links. Communicationover the network(s) 840 can be accomplished, for instance, via acommunication interface using any type of protocol, protection scheme,encoding, format, packaging, etc.

For example, the vehicle computing system 805 can communicate withexternal vehicle operator management system 850 over network(s) 840. Theexternal vehicle operator management system 850 can be remote from thevehicle computing system 805, and can provide various services such assecurity services, emergency services, roadside assistance, remotediagnostics, and/or communication services, such as with an emergencyoperator.

In some implementations, the external vehicle operator management system850 can perform analysis on logged data in order to identify ways tomaintain and/or improve a vehicle operator vigilance level. For example,machine learned models can be used to identify effective vigilanceprompts, both on an individual level (e.g., a particular vehicleoperator profile) or over a plurality of vehicle operators (e.g.,aggregated vehicle operator profiles).

In some implementations, the external vehicle operator management system850 can determine an autonomous vehicle should be operated to a safestate in which autonomous operation is disabled. For example, a vehicleoperator's vigilance level may be too low, and the external vehicleoperator management system 850 can communicate a motion plan command tothe vehicle computing system 805 in order to cause the autonomousvehicle to travel to a safe state in which autonomous operation isdisabled, such as in a parking lot. Similarly, the external vehicleoperator management system 850 can communicate a motion plan commandthat prevents a vehicle operator from causing the autonomous vehiclefrom entering or reentering an autonomous operating mode, as describedherein. For example, the autonomous vehicle may be configured to operateautonomously for one or more discrete portions of a motion plan, and atthe end of each portion, the vehicle operator may be required to engageor reengage autonomous operation before the autonomous vehicle proceedsto initiate travel in accordance with the motion plan. If the vehicleoperator vigilance level is too low, in some implementations, thevehicle operator may be prevented from causing the autonomous vehicle toenter an autonomous operating mode.

In some implementations, the external vehicle operator management system850 can assist the vehicle operator and/or remotely diagnose one or moreissues with the autonomous vehicle. For example, if a vehicle operatorvigilance level is low, a remote operator can ask the vehicle operatorif there is anything in particular affecting the vehicle operator'sability to respond to vigilance prompts. For example, external weatherconditions or traffic conditions may impair a vehicle operator's abilityto respond to vigilance prompts due to the vehicle operator paying closeattention to the conditions in which the autonomous vehicle isoperating. Similarly, a vehicle operator input device, such as amicrophone, may be malfunctioning, and the external vehicle operatormanagement system 850 may be able to remotely diagnose suchmalfunctions. In such situations, the external vehicle operatormanagement system 850 can determine that the vehicle operator'svigilance level should not prevent the vehicle operator from causing thevehicle to enter or continue operating in an autonomous operating mode.Similarly, the external vehicle operator management system 850 can resetor clear any disablement of autonomous operation, such as when theautonomous vehicle has been operated to a safe state or the vehicleoperator has been prevented from causing the vehicle to enter anautonomous operating mode.

Computing tasks, operations, and functions discussed herein as beingperformed at a vehicle (e.g., via the vehicle computing system) caninstead be performed by computing device(s) that are remote from thevehicle (e.g., via an external vehicle operator management system)and/or vice versa. Such configurations can be implemented withoutdeviating from the scope of the present disclosure. The use ofcomputer-based systems allows for a great variety of possibleconfigurations, combinations, and divisions of tasks and functionalitybetween and among components. Computer-implemented operations can beperformed on a single component or across multiple components.Computer-implemented tasks and/or operations can be performedsequentially or in parallel. Data and instructions can be stored in asingle memory device or across multiple memory devices.

The communications between computing systems described herein can occurdirectly between the systems or indirectly between the systems. Forexample, in some implementations, the computing systems can communicatevia one or more intermediary computing systems. The intermediarycomputing systems may alter the communicated data in some manner beforecommunicating it to another computing system.

The number and configuration of elements shown in the figures is notmeant to be limiting. More or less of those elements and/or differentconfigurations can be utilized in various embodiments.

While the present subject matter has been described in detail withrespect to specific example embodiments and methods thereof, it will beappreciated that those skilled in the art, upon attaining anunderstanding of the foregoing can readily produce alterations to,variations of, and equivalents to such embodiments. Accordingly, thescope of the present disclosure is by way of example rather than by wayof limitation, and the subject disclosure does not preclude inclusion ofsuch modifications, variations and/or additions to the present subjectmatter as would be readily apparent to one of ordinary skill in the art.

What is claimed is:
 1. A method for determining a vigilance level of avehicle operator present in an autonomous vehicle; the autonomousvehicle configured to operate in a fully autonomous operating mode inwhich the autonomous vehicle drives and navigates with no input from thevehicle operator, comprising: determining, by a computing systemcomprising one or more processors, a first vigilance prompt, wherein thefirst vigilance prompt is included in a plurality of vigilance prompts,wherein each of the plurality of vigilance prompts is different fromeach other vigilance prompt, wherein each vigilance prompt comprises aprompt for the vehicle operator to perform a particular interaction withthe autonomous vehicle to stimulate and engage the vehicle operatorwhile the autonomous vehicle is operating in the fully autonomousoperating mode; providing, by the computing system, the first vigilanceprompt to a vehicle operator of an autonomous vehicle while theautonomous vehicle is operating in the fully autonomous operating mode;receiving, by the computing system, a first response from the vehicleoperator in response to the first vigilance prompt while the autonomousvehicle is operating in the fully autonomous operating mode; anddetermining, by the computing system, a vehicle operator vigilance levelindicative of the vigilance of the vehicle operator based at least inpart on the first response.
 2. The method of claim 1, wherein the firstvigilance prompt comprises a visual cue displayed on a display screen ofthe autonomous vehicle or an audio cue played by a speaker of theautonomous vehicle.
 3. The method of claim 2, wherein the visual cuecomprises a color change, an icon display, or a text command.
 4. Themethod of claim 1, wherein the particular interaction comprises at leastone of a button push, a display screen touch, a shifter paddleactuation, a turn indicator actuation, a windshield wiper actuation, ora verbal response.
 5. The method of claim 1, wherein the first responsecomprises a vehicle operator interaction and a response time; andwherein determining, by the computing system, the vehicle operatorvigilance level based at least in part on the first response comprisesdetermining, by the computing system, the vehicle operator vigilancelevel based at least in part on one or more of: a determination ofwhether the response time is less than a time threshold and adetermination of whether the vehicle operator interaction is a correctinteraction.
 6. The method of claim 1, wherein, prior to determining, bythe computing system, the first vigilance prompt, the method furthercomprises: determining, by the computing system, that the vehicleoperator is ready to be provided the first vigilance prompt.
 7. Themethod of claim 6, wherein determining, by the computing system, thatthe vehicle operator is ready to be provided the first vigilance promptcomprises determining, by the computing system, that the autonomousvehicle is operating within an authorized area without any pending turnsor lane changes within a testing period.
 8. The method of claim 6,wherein determining, by the computing system, that the vehicle operatoris ready to be provided the first vigilance prompt comprisesdetermining, by the computing system, that the vehicle operator has notbeen provided a previous vigilance prompt within a threshold timeperiod.
 9. The method of claim 1, further comprising: providing, by thecomputing system, a second vigilance prompt to the vehicle operatorfollowing a time delay.
 10. The method of claim 9, wherein prior toproviding, by the computing system, the second vigilance prompt, themethod further comprises: determining, by the computing system, the timedelay based at least in part on the vehicle operator vigilance level.11. The method of claim 9, wherein prior to providing, by the computingsystem, the second vigilance prompt, the method further comprises:determining, by the computing system, a type of the second vigilanceprompt based at least in part on the vehicle operator vigilance level.12. The method of claim 9, wherein the autonomous vehicle is operatingin a testing environment, wherein the method further comprises:determining, by the computing system, a cognitive loading componentbased at least in part on the vehicle operator vigilance level; anddetermining, by the computing system, the second vigilance prompt basedat least in part on the cognitive loading component.
 13. The method ofclaim 1, further comprising: determining, by the computing system, avehicle operator management action based at least in part on the vehicleoperator vigilance level; and implementing, by the computing system, thevehicle operator management action.
 14. The method of claim 13, whereinthe vehicle operator management action comprises at least one of:determining a frequency of one or more subsequent vigilance prompts,determining a type of one or more subsequent vigilance prompts,providing the vehicle operator with vigilance level feedback, operatingthe autonomous vehicle to a safe state in which autonomous operation isdisabled, preventing the vehicle operator from causing the autonomousvehicle to enter an autonomous operating mode, contacting an externalvehicle operator management system, or logging the vehicle operatorvigilance level.
 15. An autonomous vehicle operator vigilance managementsystem, comprising; one or more processors; and one or more tangible,non-transitory, computer readable media that collectively storeinstructions that when executed by the one or more processors cause theone or more processors to perform operations comprising: determiningthat a vehicle operator present in an autonomous vehicle is ready to beprovided a first vigilance prompt, wherein the first vigilance prompt isincluded in a plurality of vigilance prompts, wherein each of theplurality of vigilance prompts is different from each other vigilanceprompt, wherein each vigilance prompt comprises a prompt for the vehicleoperator to perform a particular interaction with the autonomous vehicleto stimulate and engage the vehicle operator, wherein the autonomousvehicle is configured to operate in a fully autonomous operating mode inwhich the autonomous vehicle drives and navigates with no input from thevehicle operator; determining the first vigilance prompt; providing thefirst vigilance prompt to the vehicle operator of the autonomous vehiclewhile the autonomous vehicle is operating in the fully autonomousoperating mode; receiving a first response from the vehicle operator inresponse to the first vigilance prompt while the autonomous vehicle isoperating in the fully autonomous operating mode; and determining avehicle operator vigilance level indicative of the vigilance of thevehicle operator based at least in part on the first response.
 16. Theautonomous vehicle operator vigilance management system of claim 15,wherein determining that the vehicle operator present in the autonomousvehicle is ready to be provided the first vigilance prompt comprisesdetermining that the autonomous vehicle is operating within anauthorized area without any pending turns or lane changes within atesting period and that the vehicle operator has not been provided aprevious vigilance prompt within a threshold time period.
 17. Theautonomous vehicle operator vigilance management system of claim 15,wherein the first vigilance prompt comprises at least one of: a visualcue displayed on a display screen of the autonomous vehicle or an audiocue played by a speaker of the autonomous vehicle; and wherein theparticular interaction comprises at least one of: a button push, adisplay screen touch, a shifter paddle actuation, a turn indicatoractuation, a windshield wiper actuation, or a verbal response.
 18. Theautonomous vehicle operator vigilance management system of claim 15,wherein the operations further comprise: determining a vehicle operatormanagement action based at least in part on the first response; andimplementing the vehicle operator management action.
 19. The autonomousvehicle operator vigilance management system of claim 18, wherein thevehicle operator management action comprises at least one of:determining a frequency of one or more subsequent vigilance prompts,determining a type of one or more subsequent vigilance prompts,providing the vehicle operator with vigilance level feedback, operatingthe autonomous vehicle to a safe state in which autonomous operation isdisabled, preventing the vehicle operator from causing the autonomousvehicle to enter an autonomous operating mode, contacting an externalvehicle operator management system, or logging the vehicle operatorvigilance level.
 20. An autonomous vehicle configured to operate in atesting environment, the autonomous vehicle further configured tooperate in a fully autonomous operating mode in which the autonomousvehicle drives and navigates with no input from the vehicle operator,the autonomous vehicle comprising: a touch-sensitive display screen; aspeaker device; one or more of vehicle operator input devices, the oneor more vehicle operator input devices comprising one or more of: abutton, a shifter paddle, a turn indicator, a windshield wiper, amicrophone, or the touch-sensitive display screen; and a vehicleoperator testing normalization system, comprising: one or moreprocessors; and one or more tangible, non-transitory, computer readablemedia that collectively store instructions that when executed by the oneor more processors cause the vehicle operator testing normalizationsystem to perform operations comprising: providing a first vigilanceprompt to a vehicle operator of the autonomous vehicle via at least oneof the touch-sensitive display screen or the speaker device while theautonomous vehicle is operating in the fully autonomous operating mode,wherein the first vigilance prompt comprises a prompt for the vehicleoperator to perform a particular interaction with at least one vehicleoperator input device; receiving a first response from the vehicleoperator in response to the first vigilance prompt via the at least onevehicle operator device while the autonomous vehicle is operating in thefully autonomous operating mode, the first response comprising a vehicleoperator interaction and a response time; determining a second vigilanceprompt based at least in part on the first response, the secondvigilance prompt comprising a cognitive loading component; and providingthe second vigilance prompt to the vehicle operator of the autonomousvehicle; wherein the second vigilance prompt is determined based atleast in part on one or more of: a determination of whether the responsetime is less than a time threshold and a determination of whether thevehicle operator interaction is a correct interaction.